03 -f^-'^ 



Please type a plus sign (+) inside this box [+] pto/sb/05 (12/97) 

Approved for use through 09/30/00, OMB 0651-0032 
Patent and Trademark Office: U.S. DEPARTMENT OF COMMERCE 

Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. 



UTILITY PATENT APPLICATION TRANSMITTAL 

(Only for new nonprovisional applications under 37 CFR 1 .53(b) ^ 



Attorney Docket No. 004367, P006 Total Pages 3 -gS 
First Named Inventor or Application Identifier Daniel E. Lenoski z^:^ 



Express IVIail Label No. EL143555708US u' 



ADDRESS TO: Assistant Commissioner for Patents 

Box Patent Application 
WasFiington, D. C, 20231 



APPLICATION ELEMENTS 

See MPEP chapter 600 concerning utility patent application contents. 

1 . X Fee Transmittal Form 

(Submit an original, and a duplicate for fee processing) 



2. X Specification (Total Pages 21 ) 

(preferred arrangement set forth below) 

- Descriptive Title of the Invention 

- Cross References to Related Applications 

- Statement Regarding Fed sponsored R&D 

- Reference to Microfiche Appendix 

- Background of the Invention 

- Brief Summary of the Invention 

- Brief Description of the Drawings (if filed) 

- Detailed Description 

- Claims 

- Abstract of the Disclosure 



3. X_ Drawings(s) (35 USC 1 1 3) (Total Sheets 15 ) 

4. X_ Oath or Declaration (Total Pages 5 ) 

a. __X_ Newly Executed (Original or Copy) 

b. Copy from a Prior Application (37 CFR 1 .63(d)) 

(for Continuation/Divisional with Box 17 completed) (Note Box 5 below) 

i. DELETIONS OF INVENTOR(S^ Signed statement attached deleting 

inventor(s) named in the prior application, see 37 CFR 1 .63(d)(2) 
and 1.33(b). 

5. _ Incorporation By Reference (useable if Box 4b is checked) 

The entire disclosure of the prior application, from which a copy of the oath or 
declaration is supplied under Box 4b, is considered as being part of the 
disclosure of the accompanying application and is hereby incorporated by 
reference therein. 

6. _ Microfiche Computer Program (Appendix) 



12/01/97 



PTO/SB/OS (12/97) 



7. Nucleotide and/or Amino Acid Sequence Submission 

(if applicable, alt necessary) 

a. Computer Readable Copy 

b. Paper Copy (identical to computer copy) 

c. Statement verifying identity of above copies 

ACCOMPANYING APPLICATION PARTS 

8. X Assignment Papers (cover sheet & documents(s)) 

9. a. 37 CFR 3.73(b) Statement (where there is an assignee) 

_ b. Power of Attorney 

1 0. English Translation Document (if applicable) 

11. _ a. Information Disclosure Statement (iDS)/PT0-1 449 
_ b. Copies of IDS Citations 

12. _ Preliminary Amendment 

1 3. X_ Return Receipt Postcard (MPEP 503) (Should be specifically itemized) 

14. _ a. Small Entity Statement(s) 

b. Statement filed in prior application, Status still proper and desired 

1 5. _ Certified Copy of Priority Document(s) (if foreign priority is claimed) 

16. Other: 



17. If a CONTINUING APPLICATION, check appropriate box and supply the requisite information: 

Continuation Divisional Continuation-in-part (CIP) 

of prior application No: _ 



1 8. Correspondence Address 

Customer Number or Bar Code Label 



(Insert Customer No. or Attach Bar Code Label here) 

or 



JC Correspondence Address Below 



NAME Michael J. Mallie Reo No.36.591 

BLAKELY. SOKOLOFF. TAYLOR & ZAFMAN LLP 

ADDRESS 12400 Wilshire Boulevard 

Seventh Floor 

CITY Los Angeles STATE California ZIP CODE 90025-1026 

Country U.S.A. TELEPHONE (408) 720-8598 FAX (408) 720-9397 

1 2/01/97 . 2 - PTO/SB/05 (1 2/97) 

Approved for use through 09/30/00. 0MB 0651-0032 
Patent and Trademark Office; U.S. DEPARTMENT OF COMMERCE 



EXPRESS MAIL CERTIFICATE OF MAILING 



"Express Mail" mailing label, number: 
Date of Deposit: ?^i"iiC>0 
I hereby certify that I am causing this paper or fee to 
be deposited with the United States Postal Service 
"Express Mail Post Office to Addressee" service on 
the date Indicated above and that this paper or fee 
has been addressed to the Assistant Commissioner 
for Patenjts, Washington. D. C. 20231 _ /^ 

(Typed or printed name of person mailingj)aDer or fee) 
(Signature of person jnailing paper or fee) ^ V"^ 

':^\ n 1 do 

(Date signed) 



Filing/Issue Date: 



Serial/Patent No.: * * * * 

Client: GrOWrn NQ^-wr>T-Vo 

Title: - .^^^^'f^JBtTTTfici' fiA/TT.'|."Vm)icaTTONrc; AMp 



**** 



MAINTATNTT^n A]vTp n 



BSTZ File No.: 

Date Mailed: . 

The following has been received in the U.S. 

O Amendment/Response ( pgs.) 



004^^7, POOfi 
3/7/0 0 



Atty/Secty Initials: 
Docket Due Date: 



MJM/smo 



**** 



□ 

□ 
□ 
□ 
□ 
□ 
□ 
□ 
m 
m 
w 
□ 
m 



Appeal Brief ( pgs,) (in tnplicate) 

( ^ 1 



Application - Utility ( ^ pgs , with cover and abstract) 

Application - Rule 1.53(b) Continuation ( pgs.) 

Application - Rule 1.53(5) Divisional ( pgs.) 

Rule L53{b) CIP ( pgs.) 

Rule 1.53(d) CPA Transmittal ( pgs 

Design ( pgs.) 

PCT( pgs.) 

Provisional ( pgs.) 

Assignment and Cover Sheet ( Signed) 
Certificate of Mailing 

DecJaration&POA( 5pgs.) (Signed) 

OscksueDxs&Qg&CqiytfhyitofeSEtEdLetfcr f pg^ 
Drawings: of sheets includes 18 figures 



Application 
Application 
Application 
Application 
Application 



Patent & Trademark Office on the date stamped hereon: 

Express Mail NoEULAi5552Q 8 U£] Check No. 4 0 Q 

^ -Month(s) Extension of Time Amt: ^ g Q Q ^ Q Q 

□ itoalkHiEfelcaieStafenient&PTD- 1449 (_ pgs.) XB Check No. JA{IS3 
U Issue Fee Transmittal ' ^mt: $ 40.00 

Noace of Appeal 
Petition for Extension of Time 

Petition for 

Postcard 



□ 
□ 
□ 

XKl 
□ 
□ 
□ 
□ 
□ 



Power of Attorney ( pgs.) 

Preliminary Amendment ( pgs.) 

Reply Bnef ( pgs.) 

Response to Notice of Missmg Parts 
Small Entity Declaration for Indep. Inventor/Small Business 
XEI Transmittal Utter, X^^XX 
XH Fee Transmittal, in duplicate 



□ Other: 



04367.P006 



PATENT 



United States Patent Application 

FOR 

Distributing Fault indications and 
Maintaining and Using a Data Structure Indicating Faults To Route Traffic 

IN A Packet Switching System 

Inventors: 
Daniel E. Lenoski 

4885 Clarendon Drive, San Jose, CA 95129, a citizen of the United States 

William Eatherton 
2 1 9 North Murphy Avenue, Sunnyvale, CA 94086, a citizen of the United States 

John Andrew Fingerhut 
234 Elm Street, Apt. ill, San Mateo, C A 9440 1 , a citizen of the United States 

Jonathan S. Turner 
75 Berkshire Dr., St. Louis, MO 63 1 17, a citizen of the United States 



Prepared by: 
Blakely, Sokoloff, Taylor & Zafman llp 
12400 WiLSHiRE Boulevard 

Seventh Floor 
Los Angeles, CA 90025-1026 
408-720-3470 



EXPRESS MAIL CERTIFICATE OF MAILING 



"Express Mail" mailing label number: EL143555708US 
Date of Deposit: March 7, 2000 



I hereby certify that I am causing this paper or fee to be deposited with the United States Postal Service 
"Express Mail Post Office to Addressee"' service on the date indicated above and that this paper or fee has 
been addressed to the Commissioner of Patents and Trademarks, Washington, D. C. 20231 

Sharon M. Osofsky 

(T;^d or printed name of person mailing paper or fee) 




(Signature of person mailing paper or fee) 
(Date signed) 



04367.P006 



DISTRffiUTING FAULT INDICATIONS AND 
MAINTAINING AND USING A DATA STRUCTURE INDICATING FAULTS 
TO ROUTE TRAFFIC IN A PACKET SWITCHING SYSTEM 



FIELD OF THE INVENTION 

This invention relates to maintaining, communicating, and reacting to faults in 
packet switching systems; and more particularly, the invention relates to distributing fault 
indications and maintaining and using a data structure indicating faults used to route traffic 
10 through a packet switching system. 

BACKGROUND OF THE INVENTION 

The communications industry is rapidly changing to adjust to emerging 
technologies and ever increasing customer demand. This customer demand for new 

15 appUcations and increased performance of existing applications is driving communications 
network and system providers to employ networks and systems having greater speed and 
capacity (e.g., greater bandwidth). In trying to achieve these goals, a common approach 
taken by many communications providers is to use packet switching technology. 

As used herein, the term "packet" refers to packets of all types, including, but not 

20 limited to, fixed length cells and variable length packets. Moreover, these packets may 
contain one or more types of information, including, but not limited to, voice, data, video, 
and audio information. Furthermore, the term "system" is used generically herein to 
describe any number of components, packet switch elements, packet switches, networks, 
computer and/or communication devices or mechanisms, or combinations thereof 

25 Consumers and designers of these systems typically desire high rehability and 

increased performance at a reasonable price. A commonly used technique for helping to 
achieve this goal is for systems to provide multiple paths between a source and a 
destination. Packets of information are then dynamically routed and distributed among 
these multiple paths. It is typically more cost-effective to provide multiple slower rate links 
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or switching paths, than to provide a single higher rate path. Such designs also achieve 
other desired performance characteristics. 

It is important that packet switching systems are fault tolerant and appropriately 
detect and react to fauhs. Prior approaches to adapt to and overcome faults within a 
5 packet switch typically rely on fiill redundancy in order to enable fault-masking of failures. 
For example, such a system might include one or more extra interconnection networks 
that could become active and replace another interconnection network which has a 
detected failure. Such approaches are typically costly. Needed are new apparatus and 
methods for reacting to fauhs within a packet switching system. 

10 

SUMMARY OF THE INVENTION 

Apparatus and methods are disclosed for propagating and reacting to detected 
faults in a packet switching system. In one embodiment, a packet switching system 
15 includes muhiple input components of the packet switching system sending multiple 

packets to a multiple output components over muhiple interconnection networks. After 
the packet switching system recognizes an error within the packet switching system, the 
packet switching system notifies the multiple input components of the error. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

The appended claims set forth the features of the invention with particularity. The 
invention, together with its advantages, may be best understood fi-om the following 
detailed description taken in conjunction with the accompanying drawings of which: 

FIGs. 1 A-C are block diagrams of a few of many possible embodiments of a 
25 switching system; 

FIGs. 2A-C are block diagrams illustrating exemplary embodiments of a packet 
switching system component, such as, for example, a line card and/or input/output 
interface; 
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FIGs. 3 A-C are block diagrams of exemplary switching fabric components; 

FIG. 4 is a block diagram illustrating the operation of a broadcast mechanism 
within one of the interconnection networks for broadcasting fault information in a packet 
switching system; 

5 FIGs. 5A-B illustrate exemplary packet formats used in propagating fault 

information; 

FIGs. 6A-B illustrate exemplary data structures used to maintain fault information 
in a component of a packet switching system; 

FIGs. 7A-C are embodiments that propagate and react to detected fault conditions; 

10 and 

FIG. 8 illustrates manipulating data structures in determining a path through the 
packet switching system for a particular packet. 

DETAILED DESCRIPTION 

15 Methods and apparatus are disclosed for accumulating, distributing and reacting to 

detected faults and other error conditions in a packet switching system. Such methods and 
apparatus are not limited to a single packet switching environment. Rather, the 
architecture and functionality taught herein are extensible to an unlimited number of 
packet switching environments and embodiments in keeping with the scope and spirit of 

20 the invention. Embodiments described herein include various elements and limitations, 
with no one element or limitation contemplated as being a critical element or limitation. 
Each of the claims individually recite an aspect of the teachings disclosed herein in its 
entirety. Moreover, some embodiments described may include, inter alia, systems, 
integrated circuit chips, methods, and computer-readable medium containing instructions. 

25 The embodiments described hereinafter embody various aspects and configurations within 
the scope and spirit of the invention. 

In one embodiment, a packet switching system detects faults and propagates 
indications of these faults to the input interfaces of a packet switch. The packet switch 
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may select a route for a particular packet to avoid these faults. In this maimer, packet loss 
may be decreased upon a failure condition, and with only a short interruption delay being 
limited by the time to detect a failure and to send updates to the input sources. 

Faults are identified by various components of the packet switching system and 

5 relayed to one or more switching components to generate a broadcast packet destined for 
all input ports (e.g., to each I/O interface in a packet switch having folded input and 
output interfaces). Other embodiments generate one or more multicast or unicast packets. 
An I/O interface may be used to maintain one or more data structures indicating the state 
of various portions of the packet switching system. In one embodiment, an output 

10 availability table is maintained indicating a path over which a particular destination may be 
reached, as well as a Unk availability vector indicating which output links of the input 
interface may be currently used. Using these as masks against possible routes in a folly 
functional system, the packet switching component (e.g., I/O interface) can identify which 
routes are currently available for reaching the destination of the received packet. These 

15 routes can then be selected firom among those using numerous deterministic and non- 
deterministic methods. 

FIGs. 1 A-3C and their discussion herein are intended to provide a description of 
various exemplary packet switching systems. FIGs. 1 A-C illustrate the basic topology of 
different exemplary packet switching systems. FIG. 1 A illustrates an exemplary packet 

20 switch 100 having multiple inputs and outputs and a single interconnection network 1 10. 
FIG. IB illustrates an exemplary packet switch 140 having multiple interconnection 
networks 141 and folded input and output interfaces 149. FIG. IC illustrates an 
exemplary folded packet switch 160 having multiple interconnection networks 161 and 
folded input and output interfaces 169. Embodiments of each of these packet switches 

25 100, 140 and 160 receive, generate, accumulate, distribute, and react to detected faults in 
the manners disclosed herein. Of course, the invention is not limited to these illustrated 
operating environments and embodiments, and the packet switching systems may have 
more or less elements. 
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FIG. 1 A illustrates an exemplary embodiment of a packet switch 100. Packet 
switch 100 comprises multiple input interfaces 105, interconnection network 110, and 
output interfaces 125. Input interfaces 105 and output interfaces 125 are both coupled 
over multiple links to interconnection network 110. Line cards 101 and 131 are coupled 
5 to input interfaces 105 and output interfaces 125. In certain embodiments including other 
packet switching topologies, line cards or their functionality may be included in the packet 
switch itself, or as part of the packet switching system. 

In one embodiment, interconnection network 110 comprises multiple switch 
elements SE-1 112, SE-2 1 15, and SE-3 118 that are interconnected by multiple links. 

10 Line cards 101 and 131 may connect to other systems (not shown) to provide data items 
(e.g., packets) to be routed by packet switch 100. Fault indications may be generated, 
consumed, or processed at one or more of the line cards 101, 131, input interfaces 105, 
switch elements SE-1 112, SE-2 115, and SE-3 118, output interfaces 125, and/or other 
locations within packet switch 100 or the packet switching system. 

15 FIG. IB illustrates another exemplary operating environment and embodiment of a 

packet switch 140. Packet switch 140 comprises multiple folded input and output 
interfaces 149 interconnected over multiple links to interconnection networks 141, which 
are interconnected over multiple links returning to input and output interfaces 149. In one 
embodiment, interconnection networks 141 comprise multiple switch elements SE-1 142, 

20 SE-2 145, and SE-3 148 also intercoimected by multiple links. Interfaces 149 may 
connect via bi-directional links to line cards 139 that connect with other systems (not 
shown) to provide data items (e.g., packets) to be routed by packet switch 140. Fault 
indications may be generated, consumed, or processed at one or more of the line cards 
139, input and output interfaces 149, switch elements SE-1 142, SE-2 145, and SE-3 

25 148, and/or other locations within packet switch 140 or the packet switching system. 

FIG. IC illustrates another exemplary operating environment and embodiment of a 
packet switch 160. Packet switch 160 has a folded network topology. Packet switch 160 
comprises multiple folded input and output interfaces 169 interconnected over multiple 
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links to interconnection networks 161, which are interconnected over multiple links 
returning to interfaces 169. In one embodiment, interconnection networks 161 comprise 
multiple switch elements SE-1 & SE-3 162 and SE-2 164 also interconnected by multiple 
links. Interfaces 169 may connect via bi-directional links to line cards 159 which connect 
5 via ports 158 to other systems (not shown) to provide data items to be routed by packet 
switch 160. Fault indications may be generated, consumed, or processed at one or more of 
the line cards 159, input and output interfaces 169, switch elements SE-1 & SE-3 162 and 
SE-2 164, and/or other locations within packet switch 160 or the packet switching system. 
FIGs. 2A-C illustrate three of numerous possible embodiments of a line card, input 

10 interface, output interface, and/or input/output interface. For illustrative purposes, only 
single transmitters and receivers may be shown. It should be clear to one skilled in the art 
that multiple transmitters and receivers may be used to communicate with multiple sources 
and destinations (e.g., line cards, switch fabrics, etc.)- 

FIG. 2 A illustrates one component 220 comprising a processor 221, memory 222, 

15 storage devices 223, and one or more external interface(s) 224, and one or more packet 
switch interface(s) 225, and one or more internal communications mechanisms 229 
(shown as a bus for illustrative purposes). External interface(s) 224 transfer external 
signals with one or more communications devices or networks (e.g., one or more 
networks, including, but not limited to the Internet, intranets, private or public telephone, 

20 cellular, wireless, satellite, cable, local area, metropoUtan area and/or wide area networks). 
Memory 222 is one type of computer-readable medium, and typically comprises random 
access memory (RAM), read only memory (ROM), integrated circuits, and/or other 
memory components. Memory 222 typically stores computer-executable instructions to 
be executed by processor 221 and/or data which is manipulated by processor 221 for 

25 implementing functionality in accordance with certain embodiments of the invention. 
Storage devices 223 are another type of computer-readable medium, and typically 
comprise disk drives, diskettes, networked services, tape drives, and other storage 
devices. Storage devices 223 typically store computer-executable instructions to be 
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executed by processor 221 and/or data which is manipulated by processor 221 for 
implementing functionality an apparatus disclosed herein. Component 220 generates, 
consumes, processes and reacts to fault indications. 

As used herein, computer-readable medium is not limited to memory and storage 
5 devices; rather computer-readable medium is an extensible term including other storage 
and signaling mechanisms including interfaces and devices such as network interface cards 
and buffers therein, as well as any communications devices and signals received and 
transmitted, and other current and evolving technologies that a computerized system can 
interpret, receive, and/or transmit. 

10 FIG. 2B illustrates one component 240 having a single element providing the 

functionality of a line card and an input/output interface, for example that of line card 159 
and input/output interface 169 (FIG. IC). Figures 2B-C will be described in relation to 
FIG. IC for illustrative purposes; however, these embodiments could be used with other 
packet switch topologies and other implementations and embodiments. Component 240 

15 comprises control logic 241 implementing fiinctionality disclosed herein. In one 

embodiment, control logic 241 includes memory for storage of data and instructions. 
Control logic 241 is connected to other elements of component 240 via one or more 
internal communications mechanisms 249 (shown as a bus for illustrative purposes). 
External interface receiver 250 receives external signals, converts these signals using 

20 demultiplexor 25 1 into multiple streams of packets which are temporarily stored in 

incoming packet buffer 252. At the appropriate time, a packet is sent to the appropriate 
switch element SE-1 & SE-3 162 via transmitter to switch elements 253, Packets are 
received from switch elements SE-1 & SE-3 162 at the receiver firom switch elements 263 
and placed in the outgoing packet buffer 262. Multiplexor 261 extracts the packets and 

25 creates a multiplexed signal that is transmitted via external interface transmitter 260. 
Additionally, control logic 241 receives, generates, processes and reacts to fault 
indications as described hereinafter. 
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FIG. 2C illustrates one embodiment of a line card 270 and a switch interface 290, 
which could correspond to line card 159 and input/output interfaces 169 illustrated in 
FIG. 2C. 

Line card 270 illustrated in FIG. 2C includes control logic 271 to control 
5 operations of the input/output interface 270. Control logic 271 is connected to other 
components of line card 270 via one or more internal communications mechanisms 279 
(shown as a bus for illustrative purposes). In one embodiment, control logic 271 includes 
memory for storing instructions and data. Line card 270 also includes optional additional 
memory 272 and storage devices 273. External interface receiver 274 receives external 

10 signals 201 (FIG. 2), separates the signals into channels using demuhiplexor 275 into 
multiple streams of packets which are temporarily stored in incoming packet buffer 276. 
At the appropriate time, a packet is sent to switch interface 290 via transmitter to switch 
interface 277. Packets are received from switch interface 290 at the receiver from switch 
interface 287 and placed in the outgoing packet buffer 286. Muhiplexor 285 extracts the 

15 packets and creates a muhiplexed signal which is transmitted via external interface 

transmitter 284. In one embodiment, control logic 271, referencing a data structure within 
control logic 271 or memory 272, stores fault indications. Line card 270 may receive, 
generate, process and react to fault indications as described hereinafter. In certain 
embodiments, fault conditions may be hidden from a line card by other components which 

20 react to the fauh indications. 

The embodiment of input/output interface 290 illustrated in FIG. 2C includes 
control logic 291 implementing functionality in accordance with certain embodiments of 
the invention. Control logic 291 is connected to other components of sv^tch interface 290 
via one or more internal communications mechanisms 289 (shown as a bus for illustrative 

25 purposes). In one embodiment, control logic 291 includes memory for storing instructions 
and data. Switch interface 290 also includes optional additional memory 292 and storage 
devices 293. Line card receiver 294 receives packets from line card 270 temporarily 
stores the packets in incoming packet buffer 295. At the appropriate time, a packet is sent 
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to an appropriate switch element SE-1 & SE-3 162 via transmitter to switch elements 296. 
Packets are received from switch elements SE-1 & SE-3 162 at the receiver from switch 
elements 299 and placed in the outgoing packet buffer 298. Line card interface 
transmitter 297 then forwards these packets to line card 270, In one embodiment, control 

5 logic 291, referencing a data structure within control logic 291 or memory 292, stores 
fault indications which could be received from a line card, packet switch, or internally 
generated. Input/output interface 290 receives, generates, processes and reacts to fault 
indications as described hereinafter. 

FIGs. 3 A-C illustrate exemplary embodiments of switching elements and/or their 

10 components. FIG. 3 A is a block diagram of one embodiment of a first stage switching 
element, SE-1 300. FIG. 3B is a block diagram of one embodiment of a second stage 
switching element SE-2 330. FIG. 3C is a block diagram of one embodiment of a third 
stage switching element SE-3 360. As would be understood by one skilled in the art, the 
invention is not limited to these or any other embodiment described herein. 

15 FIG. 3 A illustrates an embodiment of SE-1 300 comprising control logic and/or 

processor 311 (hereinafter "control logic"), memory 312, storage devices 310, I/O 
interfaces 305, output queues 320, SE-2 interfaces 325, and one or more internal 
communications mechanisms 3 19 (shown as a bus for illustrative purposes). In certain 
embodiments, control logic 311 comprises custom control circuitry for controlling the 

20 operation of SE-1 300 and no storage device 3 10 is used. Memory 3 12 is one type of 

computer-readable medium, and typically comprises random access memory (RAM), read 
only memory (ROM), integrated circuits, and/or other memory components. Memory 3 12 
typically stores computer-executable instructions to be executed by control logic 311 
and/or data which is manipulated by control logic 3 1 1 for implementing fiinctionality in 

25 accordance with certain embodiments of the invention. Storage devices 3 10 are another 
type of computer-readable medium, and may comprise, for example, disk drives, diskettes, 
networked services, tape drives, and other storage devices. Storage devices 310 typically 

9 
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store computer-executable instructions to be executed by control logic 3 1 1 and/or data 
which is manipulated by control logic 3 11 for implementing functionality disclosed herein. 

SE-1 300 generates, consumes, processes and reacts to fault indications as further 
described in detail hereinafter. Briefly first, each SE-1 300 receives packets 301 and 
5 exchanges control messages 302 over one or more links with one or more input interfaces 
(not shown) such as input/output interface 290 (FIG. 2C) via I/O interfaces 305. 
Additionally, each SE-1 300 sends packets 328 and exchanges control messages 329 over 
one or more links with one or more SE-2 elements (not shown) such as SE-2 330 
(FIG. 3B) via SE-2 interfaces 325. Control logic 3 1 1 detects fauhs, generates control 

10 packets containing indications of the detected faults, and updates its fault data structure 
stored in memory 312. SE-1 300 may distribute fault indications to other packet 
switching components by sending control packets to other components as well as 
including fault indications in reserved fields of other control messages (e.g., 
acknowledgment or clear-to-send control messages) being sent. Outgoing packets and 

15 control messages are placed in output queues 320. In one embodiment, there is an output 
queue 320 for each destination, or for each class of service for each destination. 

FIG. 3B illustrates an embodiment of SE-2 330 comprising control logic and/or 
processor 341 (hereinafter "control logic"), memory 342, storage devices 340, SE-1 
interfaces 335, output queues 350, SE-3 interfaces 355, and one or more internal 

20 communications mechanisms 349 (shown as a bus for illustrative purposes). In certain 
embodiments, control logic 341 comprises custom control circuitry for controlling the 
operation of SE-2 330 and no storage device 340 is used. Memory 342 is one type of 
computer-readable medium, and typically comprises random access memory (RAM), read 
only memory (ROM), integrated circuits, and/or other memory components. Memory 342 

25 typically stores computer-executable instructions to be executed by control logic 341 
and/or data which is manipulated by control logic 341 for implementing functionality 
described herein. Storage devices 340 are another type of computer-readable medium, 
and may comprise, for example, disk drives, diskettes, networked services, tape drives, 
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and other storage devices. Storage devices 340 typically store computer-executable 
instructions to be executed by control logic 341 and/or data which is manipulated by 
control logic 341 for implementing functionality described herein. 

SE-2 330 generates, consumes, processes and reacts to fault indications as further 
5 described in detail hereinafter. Briefly first, each SE-2 330 receives packets 331 and 

exchanges control messages 332 over one or more links with one or more SE-1 elements 
(not shown) such as SE-1 300 (FIG. 3 A) via SE-1 interfaces 335. Additionally, each 
SE-2 330 sends packets 358 and exchanges control messages 359 over one or more links 
with one or more SE-3 elements (not shovm) such as SE-3 360 (FIG. 3C) via SE-3 

10 interfaces 355. In one embodiment using a folded topology, the links between (a) SE-2 
330 and SE-1 300 and (b) SE-2 330 and SE-3 360 are the same links. Control logic 341 
receives control packets containing fault indications, and updates its fault data structure 
stored in memory 342. Additionally, fault indications are broadcast through the packet 
switch or packet switching system, such as to all the I/O interfaces. Depending on the 

15 capabilities of the packet switching system, either a packet broadcast or multicast function 
could be used to eflBciently distribute the fault indications; otherwise multiple packets are 
sent. Additionally, it is possible that fault indications are collected and sent on a periodic 
or on an event basis. However, in one embodiment fault indications are distributed 
inmiediately. 

20 SE-2 330 may distribute fault indications to other packet switching components by 

sending control packets as well as including fault indications in reserved fields of other 
control messages (e.g., acknowledgment or clear-to-send control messages) being sent. 
Outgoing packets and control messages are placed in output queues 350. In one 
embodiment, there is an output queue 350 for each destination, or for each class of service 

25 for each destination. 

FIG. 3C illustrates an embodiment of SE-3 360 comprising control logic and/or 
processor 371 (hereinafter "control logic"), memory 372, storage devices 370, SE-2 
interfaces 365, output queues 380, I/O interfaces 385, and one or more internal 
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communications mechanisms 379 (shown as a bus for illustrative purposes). In certain 
embodiments, control logic 371 comprises custom control circuitry for controlling the 
operation of SE-3 360 and no storage device 370 is used. Memory 372 is one type of 
computer-readable medium, and typically comprises random access memory (RAM), read 
5 only memory (ROM), integrated circuits, and/or other memory components. Memory 372 
typically stores computer-executable instructions to be executed by control logic 371 
and/or data which is manipulated by control logic 371 for implementing functionality 
described herein. Storage devices 370 are another type of computer-readable medium, 
and may comprise, for example, disk drives, diskettes, networked services, tape drives, 

10 and other storage devices. Storage devices 370 typically store computer-executable 
instructions to be executed by control logic 371 and/or data which is manipulated by 
control logic 371 for implementing functionality described herein. 

SE-3 360 generates, consumes, processes and reacts to fault indications as further 
described in detail hereinafler. Briefly first, each SE-3 360 receives packets 361 and 

15 exchanges control messages 362 over one or more links with one or more SE-2 elements 
(not shown) such as SE-2 330 (FIG. 3B) via SE-2 interfaces 365. Additionally, SE-3 360 
sends packets 388 and exchanges control messages 389 over one or more links with one 
or more output interface elements (not shown) such as Input/Output interface 390 (FIG. 
2C) via I/O interfaces 385. Control logic 371 receives control packets containing fault 

20 indications, and updates its fauh data structure stored in memory 372. SE-3 360 may 
distribute fault indications to other packet switching components by sending control 
packets as well as including fault indications in reserved fields of other control messages 
(e.g., acknowledgment or ciear-to-send control messages) being sent. Outgoing packets 
and control messages are placed in output queues 380. In one embodiment, there is an 

25 output queue 380 for each destination, or for each class of service for each destination. 
FIG. 4 illustrates a logical diagram of the operation of an embodiment for 
distributing fault indications to I/O interfaces 410. In certain embodiments of packet 
switching systems, it is important to broadcast the status of detected faults to all I/O 
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interfaces 410. FIG. 4 illustrates the operation of the collection and broadcast of an 
indication of an identified fault using a packet switching system having multiple line cards 
401, each connected to an 1/0 mterface 410. Note, the topology illustrated in FIG, 4 is 
that of a folded packet switch, and that each line card 401 and I/O interface 410 are shown 
5 both on the left and right side of FIG. 4 for simplicity of illustration. Also, switch 
elements SE-1 411 and SE-3 413 are illustrated separately; however in certain 
embodiments such as that illustrated in FIG. IC, these are embodied in the same 
component. 

For illustrative purposes, the example shown in FIG 4 assumes a fault is detected 

10 in SE-1 component 41 IB. The teachings of FIG. 4 and its discussion can be directly 

applied to other components. Upon detection of a fault, SE-1 component 41 IB generates 
a control packet containing the fault indication. An example of such is shown by packet 
500 in FIG. 5 A, whose header contains the destination address of a broadcast mechanism 
425, and contains the indication of the fault in field 502. This control packet 500 is sent 

1 5 by SE-1 component 41 IB to broadcast mechanism 425 over link 441F. Broadcast 

mechanism 425 receives packet 500, and then creates and sends one or more packets to 
the I/O interfaces. An example of such is shown by packet 5 10 in FIG. 5B. Header field 
511 contains the destination address of an I/O Interface 410, and field 512 contains the 
indication of the fault. One or more packets 510 are then sent over links 429 to each of 

20 the I/O interfaces 410. 

The processing by one embodiment of a broadcast mechanism is illustrated in 
FIG. 7A. Referring to FIG. 7A, processing begins at step 700, and proceeds to step 702 
where a control packet containing a fault indication is received. Then, in step 704, one or 
more control packets are created and broadcast, multicast, and/or sent to each of the I/O 

25 interfaces. 

In one embodiment, a control packet containing a fault indication is sent to two or 
more different broadcast mechanisms to increase the likelihood of all I/O interfaces 
receiving the fault indication. As would be understood by one skilled in the art, these and 
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other variations are contemplated and accommodated by the extensive number of possible 
embodiments. 

FIG. 7B illustrates one embodiment of the processing by an I/O interface for 
maintaining one or more data structures containing fault indications. Referring to FIG. 7B, 
5 processing begins at step 720, and proceeds to step 722 where the I/O interface receives a 
packet containing a fault indication. Next, in step 724, the I/O interface updates one or 
more data structures it maintains of fault indications. These data structures may be 
updated to directly indicate the received indications of faults. In some embodiments, an 
additional thresholding function is performed to disable traffic from being sent to a 

1 0 destination w^hen the number of paths leading to the destination falls below some 

predetermined threshold value (e.g., number of paths, percentage of total paths, etc.). 
This predetermined threshold value may be preconfigured, or determined during operation 
to adjust to conditions of the packet switching system. For example, if nine of ten paths 
leading to a destination are unavailable, then traffic being sent to the destination may be 

15 required to be sent over some smaller number of paths leading to the destination than 

when the packet switching system is fully operational This may lead to congestion inside 
the packet switch system and/or congestion at the input interfaces of the packet switch 
system. By disabling the traffic to the destination fi-om some or all of the input interfaces, 
this traffic congestion situation may be avoided. Processing then returns to step 722 to 

20 receive more fault indications. 

FIGs, 6A-B illustrate two of many different embodiments of data structures that 
could be maintained by an I/O interface to indicate fault conditions within the packet 
switching system. Data structure 600 is a table, typically implemented as a 
two-dimensional array or linked list, which maintains an entry for each output component 

25 602 that is reachable over each interconnection network 601. In this manner, an I/O 
interface can easily determine which interconnection networks are available to deliver a 
particular packet destined for a particular output component (e.g., an I/O interface, line 
card, or port thereof, etc.) by looking at the appropriate colunrn of data structure 600. 
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In one embodiment, data structure 610 is maintained in place of, or in addition to, 
data structure 600. One embodiment of data structure 610 is a link status vector 612, 
typically implemented as a bit vector, which indicates a link status for each interconnection 
network 61 1 to which it is coupled. In one embodiment, a bit is cleared in data structure 
5 610 if the I/O interface is prevented from sending to the particular interconnection 
network 611. 

FIG. 8 illustrates one embodiment's use of data structures 600 and 610 in 
determining a route for a particular packet. A set of possible routing paths is determined 
and maintained by a routing mechanism, which can be represented by bit vector 810. By 

10 performing an AND function with output availability table 820 (e.g., the colunm of data 
structure 600 corresponding to the particular destination) and link status vector 830 (e.g., 
link status vector 612), a set of possible paths for the particular packet is generated and 
represented by bit vector 840. Bit vector 840 is then used by a routing mechanism to 
select one of the possible paths (e.g., corresponding to a set bit in bit vector 840), such as 

15 by using a round-robin, random, even distribution, or other deterministic or 

non-deterministic method. In some embodiments, as previously described herein, a 
thresholding function is used to set the values corresponding to a particular output of the 
output availability table 820 to zero when the number of available paths represented in the 
routing paths data structure 810 falls below a predetermined threshold value. 

20 FIG. 7C illustrates one embodiment of an I/O interface for determining a route for, 

and sending packets using the maintained one or more data structures of fault indications. 
Processing begins at step 740, and proceeds to step 742 where a particular packet is 
received to be sent to a particular destination. Next, in step 744, a routing mechanism 
determines a particular route for the received packet from those routes indicated as 

25 available in the maintained one or more data structures. One embodiment of this 
processing was previously described in relation to FIG. 8, 

Next, if there is a possible route for the received packet, as determined in step 746, 
then the packet is sent over the route determined in step 744. Otherwise, in step 750, the 
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packet may be dropped packet as the packet cannot be currently sent from the I/O 
interface to its intended destination. Additionally, some error handling (e.g., generating 
alarms, dropping the packet, sending control information to the packet's sender, etc) may 
be performed in step 750. Processing then returns to step 742 to process more packets. 
5 In view of the many possible embodiments to which the principles of our invention 

may be applied, it will be appreciated that the embodiments and aspects thereof described 
herein with respect to the drawings/figures are only illustrative and should not be taken as 
limiting the scope of the invention. To the contrary, the invention as described herein 
contemplates all such embodiments as may come within the scope of the following claims 
10 and equivalents thereof 
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CLAIMS 

What is claimed is: 

1. A method performed by a packet switching system, the method comprising: 

a plurality of input components of the packet switching system sending a plurality 
5 of packets to a plurality of output components over a plurality of interconnection 
networks; 

the packet switching system recognizing an error within the packet switching 
system; and 

the packet switching system notifying the plurality of input components of the 

10 error. 

2. The method of claim 1, wherein notifying the plurality of input components 
includes sending a packet containing an indication of the error to each of the plurality of 
input components. 

3. The method of claim 1, wherein notifying the plurality of input components 
15 includes sending a packet containing an indication of the error. 

4. The method of claim 3, wherein notifying the plurality of input components 
includes sending a packet to a broadcast component of the packet switching system, and 
further comprising the broadcast component broadcasting a status notification packet 
containing an indication of the error to the plurality of input components. 

20 5 . The method of claim 4, wherein notifying the plurality of input components 

includes sending a second packet to a second broadcast component of the packet 
switching system, and further comprising the second broadcast component broadcasting a 
second status notification packet containing a second indication of the error to the 
plurality of input components 
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6. The method of claim 1, further comprising each of the plurality of input 
components updating one or more status data structures in response to receiving a 
notification of the error. 

7. The method of claim 6, forther comprising each of the plurality of input 

5 components determining which of a plurality of paths leading to a destination output 

component over which to send a particular packet, the path determined by referencing the 
one or more status data structures. 

8. The method of claim 6, wherein each of the plurality of input component 
references its one or more status data structures in determining which of a plurality of 

10 paths leading to a destination output component over which to send a particular packet. 

9. The method of claim 6, wherein the one or more data structures include an 
output availability table to indicate whether a possible path through the packet switching 
system jfrom the input component to a particular destination is available. 

10. The method of claim 6, further comprising disabling at least one of the plurality 
15 of input components fi'om sending packets to a particular destination of the packet 

switching system when a number of possible paths through the packet switching system 
leading to a particular destination falls below a predetermined threshold value, 

1 1 . The method of claim 6, wherein the one or more data structures include a fault 
indication for a first output component over a first interconnection network of the plurality 

20 of interconnection networks, and further comprising sending a first packet over the first 
interconnection network to a second output component. 

12. The method of claim 6, wherein the one or more data structures include a link 
availability table to indicate which of a plurality of outputs of a particular input component 
are available. 
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13, A packet sAvitching system comprising: 

a plurality of input components, each of the plurality of input components 
maintaining a fault data structure; 

a plurality of output components; and 
5 a plurality of interconnection networks, each of the plurality of interconnection 

networks coupled to each of the plurality of input components and to each of the plurality 
of the output components to provide a plurality of paths between each of the plurality of 
input components and the plurality of output components; 

wherein the fauh data structure of at least one of the plurality of input components 
10 includes an indication of which interconnection networks the at least one input component 
may send packets through to reach a particular output component. 

14. The packet switching system of claim 13, further comprising a broadcast 
mechanism to receive an indication of a problem within the packet switching system and to 
notify the plurality of input components of the problem. 

15 15. The packet switching system of claim 14, wherein the broadcast mechanism is 

located in one of the plurality of interconnection networks. 

16. The packet switching system of claim 14, wherein the broadcast mechanism is 
located in each of the plurality of interconnection networks. 

17. The packet switching system of claim 13, wherein each of the input 

20 components references its associated fault data structure in determining which of the 
plurality of interconnection network through which to send a particular packet, 

18. The packet switching system of claim 13, vv^herein the fault data structure 
includes an output availability indication of which of the plurality of interconnection 
networks through which its associated input component may send packets. 

25 
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ABSTRACT 

Methods and apparatus are disclosed for distributing fault indications and 
maintaining and using a data structure indicating faults to route traffic in a packet 
switching system. In one embodiment, a packet switching system detects faults and 
propagates indications of these faults to the input interfaces of a packet switch, so the 
packet switch can adapt the selection of a route over which to send a particular packet. 
Faults are identified by various components of the packet switching system and relayed to 
one or more switching components to generate a broadcast packet destined for all input 
ports (i.e., to each I/O interface in a packet switch having folded input and output 
interfaces). Other embodiments, generate one or more multicast or unicast packets. The 
I/O interface maintains one or more data structures indicating the state of various portions 
of the packet switching system. In one embodiment, an output availability table is 
maintained indicating over which path a particular destination may be reached, as well as a 
link availability vector indicating which output likes of the input interface may be currently 
used. Using these as masks against possible routes in a fiiUy functional system, the packet 
switching component (e.g., I/O interface) can identify which routes are currently available 
for reaching the destination of the received packet. These routes can then be selected 
between using one of numerous deterministic and non-deterministic methods. 
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44,587; Andrew C. Chen, Reg. No. 43,544; Thomas M. Coester, Reg. No. 39,637; Alin Corie, Reg. No. 
P46,244; Dennis M. deGuzman, Reg. No. 41,702; Stephen M. De Klerk, under 37 C.F.R. § 10.9(b); 
Michael Anthony DeSanctis, Reg. No. 39,957; Daniel M. De Vos, Reg. No. 37,813; Robert Andrew Diehl, 
Reg. No. 40,992; Sanjeet Dutta, Reg. No. P46,145; Matthew C. Fagan, Reg. No. 37,542; Tarek N. Fahmi, 
Reg. No. 41,402; Paramita Ghosh, Reg. No. 42,806; James Y. Go, Reg. No. 40,621; James A. Henry, 
Reg. No. 41,064; Willmore F. Holbrow 111, Reg. No. P41,845; Sheryl Sue Holloway, Reg. No. 37,850; 
George W Hoover II, Reg. No. 32,992; Eric S. Hyman, Reg. No. 30,139; William W. Kidd, Reg. No. 
31,772; Sang Hui Kim, Reg. No. 40,450; Eric T. King, Reg. No. 44,188; Erica W. Kuo, Reg. No. 42,775; 
Kurt P. Leyendecker, Reg. No. 42,799; Michael J. Mallie, Reg. No. 36,591; Andre L Marais, under 37 
C.F.R. § 10.9(b); Paul A. Mendonsa, Reg. No. 42,879; Darren J. Milliken, Reg. 42,004; Lisa A. Norris, 
Reg. No. 44,976; Chun M. Ng, Reg. No. 36,878; Thien T. Nguyen, Reg. No. 43,835; Thinh V. Nguyen, 
Reg. No. 42,034; Dennis A. Nicholls, Reg. No. 42,036; Daniel E. Ovanezian, Reg. No. 41,236; Marina 
Portnova, Reg. No. P45,750; Babak Redjaian, Reg. No. 42,096; William F. Ryann, Reg. 44,313; James 
H. Salter, Reg. No. 35,668; William W. Schaal, Reg. No. 39,018; James C. Scheller, Reg. No. 31,195; 
Jeffrey Sam Smith, Reg. No. 39,377; Maria McCormack Sobrino, Reg. No. 31,639; Stanley W. Sokoloff, 
Reg. No. 25,128; Judith A. Szepesi, Reg. No. 39,393; Vincent P. Tassinari, Reg. No. 42,179; Edwin H. 
Taylor, Reg. No. 25,129; John F. Travis, Reg. No. 43,203; George G. C. Tseng, Reg. No. 41,355; Joseph 
A. Twarowski, Reg. No. 42,191; Lester J. Vincent, Reg. No. 31,460; Glenn E. Von Tersch, Reg. No. 
41,364; John Patrick Ward, Reg. No. 40,216; Mark L Watson, Reg. No. P46,322; Thomas C. Webster, 
Reg. No. P46,154; Charles T. J. Weigell, Reg. No. 43,398; Kirk D. Williams, Reg. No. 42,229; James M. 
Wu, Reg. No. 45,241; Steven D. Yates, Reg. No. 42,242; and Norman Zafman, Reg. No. 26,250; my 
patent attorneys, and Justin M. Dillon, Reg. No. 42,486; my patent agent, of BLAKELY, SOKOLOFF, 
TAYLOR & ZAFMAN LLP, with offices located at 12400 Wilshire Boulevard, 7th Floor, Los Angeles, 
California 90025, telephone (310) 207-3800, and James R. Thein, Reg. No. 31,710, my patent attorney. 
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APPENDIX B 



Title 37, Code of Federal Regulations, Section 1 .56 
Duty to Disclose Information Material to Patentability 

(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the most effective patent examination occurs when, at the time an application is being examined, the 
Office is aware of and evaluates the teachings of all information materia! to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office all information known to that individual 
to be material to patentability as defined in this section. The duty to disclosure information exists with respect 
to each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the information is not material to the patentability of any claim 
remaining under consideration in the application. There is no duty to submit information which is not material 
to the patentability of any existing claim. The duty to disclosure all information known to be materia! to 
patentability is deemed to be satisfied if all information known to be material to patentability of any claim 
issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§1 .97(b)-(d) 
and 1.98, However, no patent will be granted on an application in connection with which fraud on the Office 
was practiced or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. 
The Office encourages applicants to carefully examine: 

(1 ) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material information 
contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record or being made or record in the application, and 

(1 ) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(il) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the information compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim 
its broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1 ) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively Involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing information to the attorney, agent, or inventor. 
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